Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add e2e test for visual prompting #2360

Merged
merged 32 commits into from
Jul 18, 2023
Merged

Add e2e test for visual prompting #2360

merged 32 commits into from
Jul 18, 2023

Conversation

sungchul2
Copy link
Contributor

@sungchul2 sungchul2 commented Jul 14, 2023 •
edited
Loading

Summary

This PR includes:

  • e2e test
  • enable pip install -e .[visual_prompting]
  • enable tox devenv venv/otx -e tests-visprompt-py310
  • remove unused configs
  • fix about no returning the first label in VisualPromptingDatasetAdapter

Issues

  • Due to using small samples (car_tree_bug), statistics seem unstable and occurred data.size() == OC during otx optimize. Solve by limiting activation range.
    → Local e2e was passed, but WGISD occurred the error data.size() == OC. PTQ for visual prompting looks like unstable, so it is temporarily skipped just for now.
f32 u8 (image encoder only)
car_tree_bug 0.8329 0.8406
Trashcan 0.8147 0.8172
FLARE22 0.8951 0.8883
WGISD 0.9425 error

How to test

# run e2e
$ tox -vv -e tests-all-py310 -- tests/e2e/cli/visual_prompting/test_visual_prompting.py
======================================================== 7 passed, 1 warning in 171.07s (0:02:51) =========================================================

# pip install -e .[visual_prompting]
$ python -m venv venv
$ source venv/bin/activate
$ (venv) pip install -e .[visual_prompting]
$ (venv) pytest tests/e2e/cli/visual_prompting/test_visual_prompting.py
======================================================== 7 passed, 1 warning in 153.34s (0:02:33) =========================================================

# tox devenv venv/otx -e tests-visprompt-py310
$ tox devenv venv/otx -e tests-visprompt-py310
$ source venv/otx/bin/activate
$ tox -vv -e tests-visprompt-py310 -- tests/e2e/cli/visual_prompting/test_visual_prompting.py
======================================================== 7 passed, 1 warning in 168.64s (0:02:48) ========================================================

# integration test
$ tox -e tests-visprompt-py310 -- tests/integration/cli/visual_prompting
==================================================== 8 passed, 2 skipped, 1 warning in 93.69s (0:01:33) ====================================================

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@sungchul2 sungchul2 added this to the 1.4.0 milestone Jul 14, 2023
@github-actions github-actions bot added DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM ALGO Any changes in OTX Algo Tasks implementation TEST Any changes in tests BUILD labels Jul 14, 2023
@sungchul2 sungchul2 marked this pull request as ready for review July 14, 2023 09:20
@sungchul2 sungchul2 requested a review from a team as a code owner July 14, 2023 09:20
@sungchul2 sungchul2 requested a review from wonjuleee July 14, 2023 09:21
@sungchul2 sungchul2 requested a review from yunchu July 17, 2023 02:12
@yunchu
Copy link
Contributor

yunchu commented Jul 17, 2023

@sungchul2 can you confirm that the integration tests are passed as well on you local env?
I think it would be done by command tox -e tests-vp-py310 -- tests/integration/cli/visual_prompting

@sungchul2
Copy link
Contributor Author

sungchul2 commented Jul 17, 2023 •
edited
Loading

@sungchul2 can you confirm that the integration tests are passed as well on you local env? I think it would be done by command tox -e tests-vp-py310 -- tests/integration/cli/visual_prompting

vp -> visprompt because vp seems too short.

$ tox -e tests-visprompt-py310 -- tests/integration/cli/visual_prompting
==================================================== 8 passed, 2 skipped, 1 warning in 93.69s (0:01:33) ====================================================

sungmanc
sungmanc previously approved these changes Jul 17, 2023
View reviewed changes
Copy link
Contributor

@sungmanc sungmanc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yunchu
Copy link
Contributor

yunchu commented Jul 17, 2023

@sungchul2 can you clarify that why e2e testing results for visual prompting show different number of passed/skipped in your PR description?
image

@sungchul2
Copy link
Contributor Author

@sungchul2 can you clarify that why e2e testing results for visual prompting show different number of passed/skipped in your PR description? image

Updated.

yunchu
yunchu approved these changes Jul 17, 2023
View reviewed changes
Copy link
Contributor

@yunchu yunchu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sungchul2 sungchul2 enabled auto-merge (squash) July 18, 2023 00:24
@sungchul2 sungchul2 merged commit edd1a8c into openvinotoolkit:releases/1.4.0 Jul 18, 2023
@sungchul2 sungchul2 deleted the visual-prompting-e2e branch July 18, 2023 01:03
yunchu added a commit that referenced this pull request Jul 24, 2023
@yunchu @sungchul2
@vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @sovrasov @GalyaZalesskaya @harimkang
Merge back 1.4.0 to develop (#2387)
6662f48 
* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

* Fix semantic segmentation soft prediction dtype (#2322)

* Contrain yapf verison lesser than 0.40.0 (#2328)

* Fix detection e2e tests (#2327)

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* OTX deploy for visual prompting task  (#2311)

* Bump albumentations version in anomaly requirements (#2350)

* Update action detection (#2346)

* Add OTX optimize for visual prompting task (#2318)

* Update detection docs (#2335)

* Add visual prompting documentation (#2354)

* Remove custom modelapi patch in visual prompting (#2359)

* Fix graph metric order and label issues (#2356)

* Update multi-label document and conversion script (#2358)

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* Fixing detection saliency map for one class case (#2368)

* Add e2e test for visual prompting (#2360)

* Fix e2e (#2366)

* Add Dino head unit tests (#2344)

* Update for release 1.4.0rc2 (#2370)

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix H-label classification (#2377)

* Update for release 1.4 (#2380)

* Switch to PTQ for sseg (#2374)

* Fix invalid import structures in otx.api (#2383)

* Update for 1.4.0rc4 (#2385)

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: Vladislav Sovrasov <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
sovrasov added a commit that referenced this pull request Nov 27, 2023
@sovrasov @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @GalyaZalesskaya @harimkang @ashwinvaidya17
Update Intel devices branch to the latest develop state (#2666)
2239ac1 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <[email protected]>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <[email protected]>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <[email protected]>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <[email protected]>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <[email protected]>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <[email protected]>

* Prevent division by zero

Signed-off-by: Songki Choi <[email protected]>

* Update license

Signed-off-by: Songki Choi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <[email protected]>

* Fix inefficient sampling

Signed-off-by: Songki Choi <[email protected]>

* Revert indexing

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

---------

Signed-off-by: Songki Choi <[email protected]>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <[email protected]>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <[email protected]>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Yunchu Lee <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: sungmanc <[email protected]>
sovrasov added a commit that referenced this pull request Nov 30, 2023
@sovrasov @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @GalyaZalesskaya @harimkang @ashwinvaidya17
Add XPU support to anomaly task (#2677)
c1beb05 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <[email protected]>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <[email protected]>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <[email protected]>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <[email protected]>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <[email protected]>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <[email protected]>

* Prevent division by zero

Signed-off-by: Songki Choi <[email protected]>

* Update license

Signed-off-by: Songki Choi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <[email protected]>

* Fix inefficient sampling

Signed-off-by: Songki Choi <[email protected]>

* Revert indexing

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

---------

Signed-off-by: Songki Choi <[email protected]>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <[email protected]>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <[email protected]>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

* Add xpu accelerator

* Tmp patch for anomaly trainer

* Use XPU callback for anomaly training

* Update xpu accelerator

* Fix for anomaly xpu callback

* Fix validation batch logic

* Cleanup, add docstrings

* Refine xpu callback

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Yunchu Lee <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: sungmanc <[email protected]>
sovrasov added a commit that referenced this pull request Dec 14, 2023
@eunwoosh @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @sovrasov @GalyaZalesskaya @harimkang @ashwinvaidya17
Merge develop to develop-idev (#2727)
927fab6 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <[email protected]>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <[email protected]>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <[email protected]>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <[email protected]>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <[email protected]>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <[email protected]>

* Prevent division by zero

Signed-off-by: Songki Choi <[email protected]>

* Update license

Signed-off-by: Songki Choi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <[email protected]>

* Fix inefficient sampling

Signed-off-by: Songki Choi <[email protected]>

* Revert indexing

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

---------

Signed-off-by: Songki Choi <[email protected]>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <[email protected]>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Make `max_num_detections` configurable (#2647)

* Make max_num_detections configurable

* Fix RCNN case with integration test

* Apply max_num_detections to train_cfg, too

---------
Signed-off-by: Songki Choi <[email protected]>

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <[email protected]>

* Fix CPU training issue on non-CUDA system (#2655)

Fix bug that auto adaptive batch size raises an error if CUDA isn't available (#2410)

---------
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update docs for enabling sphinx.ext.autosummary (#2654)

*  fix some errors/warnings on docs source

* enable sphinx-autosummary for API reference documentation

* Update Makefile

* update sphinx configuration

* Update PTQ docs (#2672)

* Replace POT -> PTQ

* Fixes from comments

* Update regression tests for develop (#2652)

* Update regression tests (#2556)

* update reg tests

* update test suit

* update regression criteria

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Exclude py37 target config for cibuildwheel (#2673)

* Add `--dryrun` option to tools/experiment.py (#2674)

* Fix variable override bug

* Add --dryrun option to see experiment list

---------
Signed-off-by: Songki Choi <[email protected]>

* Update OTX explain CLI arguments (#2671)

* Change int8 to uint8 to XAI tests

* Add probabilities for CLI demo

* Rename arguments for explain

* Fix pre-commit

* Remove extra changes

* Fix integration tests

* Fix integration "explain_all_classes" test for OV

* Fix e2e tests for explain (#2681)

* Add README.md for experiment.py (#2688)

* write draft readme

* refine readme

* align with pre-commit

* Fix typo in reg test cmd (#2691)

* Select more proper model weight file according to commands run just before (#2696)

* consider more complex case when prepare eval and optimize

* update readme

* align with pre-commit

* add comment

* Add visual prompting zero-shot learning (`learn` & `infer`) (#2616)

* Add algobackend & temp configs

* Update config

* WIP

* Fix to enable `algo_backend`

* (WIP) Update dataset

* (WIP) Update configs

* (WIP) Update tasks

* (WIP) Update models

* Enable `learn` task through otx.train

* (WIP) enable `infer` (TODO : normalize points)

* Fix when `state_dict` is None

* Enable `ZeroShotInferenceCallback`

* Enable otx infer

* Enable to independently use processor

* Revert max_steps

* Change `postprocess_masks` to `staticmethod`

* Add `PromptGetter` & Enable `learn` and `infer`

* precommit

* Fix args

* Fix typo

* Change `id` to `id_`

* Fix import

* Fix args

* precommit

* (WIP) Add unit tests

* Fix

* Add unit tests

* Fix

* Add integration tests

* precommit

* Update CHANGELOG.md

* Update docstring and type annotations

* Fix

* precommit

* Fix unused args

* precommit

* Fix

* Fix unsupported dtype in ov graph constant converter (#2676)

* Fix unsupported dtype in ov graph constant converter

* Fix more ov-graph related unit tests

* Skip failure TC with adding issue number ref. (#2717)

* Fix visual prompting e2e test (#2719)

Skip zero-shot e2e

* Remove duplicated variable combination in experiment.py (#2713)

* Enhance detection & instance segmentation experiment (#2710)

* Compute precision and recall along with f-measure

* Log performance

* Accept ellipse annotation from datumaro format

* Fix dataset adapter condition for det/iset

* Insert garbage collection btw experiments

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Yunchu Lee <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: Vladislav Sovrasov <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: sungmanc <[email protected]>
sovrasov added a commit that referenced this pull request Feb 20, 2024
@kprokofi @yunchu
@sungchul2 @vinnamkim @negvet @goodsong81 @eunwoosh @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @sovrasov @GalyaZalesskaya @harimkang @ashwinvaidya17
Merge develop 2 (#2936)
6914551 
* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - tox-dev/tox#3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <[email protected]>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <[email protected]>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <[email protected]>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <[email protected]>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <[email protected]>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <[email protected]>

* Prevent division by zero

Signed-off-by: Songki Choi <[email protected]>

* Update license

Signed-off-by: Songki Choi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <[email protected]>

* Fix inefficient sampling

Signed-off-by: Songki Choi <[email protected]>

* Revert indexing

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

---------

Signed-off-by: Songki Choi <[email protected]>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <[email protected]>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Make `max_num_detections` configurable (#2647)

* Make max_num_detections configurable

* Fix RCNN case with integration test

* Apply max_num_detections to train_cfg, too

---------
Signed-off-by: Songki Choi <[email protected]>

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <[email protected]>

* Fix CPU training issue on non-CUDA system (#2655)

Fix bug that auto adaptive batch size raises an error if CUDA isn't available (#2410)

---------
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update docs for enabling sphinx.ext.autosummary (#2654)

*  fix some errors/warnings on docs source

* enable sphinx-autosummary for API reference documentation

* Update Makefile

* update sphinx configuration

* Update PTQ docs (#2672)

* Replace POT -> PTQ

* Fixes from comments

* Update regression tests for develop (#2652)

* Update regression tests (#2556)

* update reg tests

* update test suit

* update regression criteria

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Exclude py37 target config for cibuildwheel (#2673)

* Add `--dryrun` option to tools/experiment.py (#2674)

* Fix variable override bug

* Add --dryrun option to see experiment list

---------
Signed-off-by: Songki Choi <[email protected]>

* Update OTX explain CLI arguments (#2671)

* Change int8 to uint8 to XAI tests

* Add probabilities for CLI demo

* Rename arguments for explain

* Fix pre-commit

* Remove extra changes

* Fix integration tests

* Fix integration "explain_all_classes" test for OV

* Fix e2e tests for explain (#2681)

* Add README.md for experiment.py (#2688)

* write draft readme

* refine readme

* align with pre-commit

* Fix typo in reg test cmd (#2691)

* Select more proper model weight file according to commands run just before (#2696)

* consider more complex case when prepare eval and optimize

* update readme

* align with pre-commit

* add comment

* Add visual prompting zero-shot learning (`learn` & `infer`) (#2616)

* Add algobackend & temp configs

* Update config

* WIP

* Fix to enable `algo_backend`

* (WIP) Update dataset

* (WIP) Update configs

* (WIP) Update tasks

* (WIP) Update models

* Enable `learn` task through otx.train

* (WIP) enable `infer` (TODO : normalize points)

* Fix when `state_dict` is None

* Enable `ZeroShotInferenceCallback`

* Enable otx infer

* Enable to independently use processor

* Revert max_steps

* Change `postprocess_masks` to `staticmethod`

* Add `PromptGetter` & Enable `learn` and `infer`

* precommit

* Fix args

* Fix typo

* Change `id` to `id_`

* Fix import

* Fix args

* precommit

* (WIP) Add unit tests

* Fix

* Add unit tests

* Fix

* Add integration tests

* precommit

* Update CHANGELOG.md

* Update docstring and type annotations

* Fix

* precommit

* Fix unused args

* precommit

* Fix

* Fix unsupported dtype in ov graph constant converter (#2676)

* Fix unsupported dtype in ov graph constant converter

* Fix more ov-graph related unit tests

* Skip failure TC with adding issue number ref. (#2717)

* Fix visual prompting e2e test (#2719)

Skip zero-shot e2e

* Remove duplicated variable combination in experiment.py (#2713)

* Enhance detection & instance segmentation experiment (#2710)

* Compute precision and recall along with f-measure

* Log performance

* Accept ellipse annotation from datumaro format

* Fix dataset adapter condition for det/iset

* Insert garbage collection btw experiments

* Upgrade NNCF & OpenVINO (#2656)

* Upgrade OV MAPI and NNCF version

* Update demo requirements

* Update changelog

* Update datumaro

* Add rust installation

* Update NNCF configs for IS models

* Update more fqs

* Exclude nncf from upgrade

* Revert "Update NNCF configs for IS models"

This reverts commit 7c8db8c.

* Revert "Update more fqs"

This reverts commit 5b91c32.

* Revert "Exclude nncf from upgrade"

This reverts commit 8926c51.

* Update FQs

* Revert "Revert "Update NNCF configs for IS models""

This reverts commit f904c0c.

* Disable explain for NNCF detection task

* Update FQs for anomaly

* Update cls FQs

* Update datumaro

* Update exportable code requirements

* Add unit test to cover the changes

* Fix multilabel classification class index (#2736)

Fix multilabel cls

* Refine parsing final score of training in experiment.py (#2738)

refine val parser

* Make mean teacher algorithm consider distributed training (#2729)

* make mean_teacher consider distributed training

* align with pre-commit

* re-enable test case

* move tensor not to cuda but current device

* apply comment

* Add visual prompting zero-shot learning (`export`, IR inference) (#2706)

* Add algobackend & temp configs

* Update config

* WIP

* Fix to enable `algo_backend`

* (WIP) Update dataset

* (WIP) Update configs

* (WIP) Update tasks

* (WIP) Update models

* Enable `learn` task through otx.train

* (WIP) enable `infer` (TODO : normalize points)

* Fix when `state_dict` is None

* Enable `ZeroShotInferenceCallback`

* Enable otx infer

* Enable to independently use processor

* Revert max_steps

* Change `postprocess_masks` to `staticmethod`

* Add `PromptGetter` & Enable `learn` and `infer`

* precommit

* Fix args

* Fix typo

* Change `id` to `id_`

* Fix import

* Fix args

* precommit

* (WIP) Add unit tests

* Fix

* Add unit tests

* Fix

* Add integration tests

* precommit

* Update CHANGELOG.md

* Update docstring and type annotations

* Fix

* precommit

* Reuse SAM modules for `export` & Add dataset

* Fix

* Enable `export`

* Convert fp32

* Update logic & tests

* Fix & Add prompt getter in `model_adapter_keys`

* Initial `Inferencer`, `Task`, and `Model`

* Fix to use original mask decoder during inference

* Remove internal loop in `PromptGetter`

* Update IO

* (WIP) Add unit tests for export

* Update `PromptGetter` to use only tensor ops

* Fix issue about `original_size` disappear in onnx graph

* (WIP) Add export unit test

* Update

* Fix typo

* Update

* Fix unexpected IF & Update inputs to avoid issues which OV on CPU doesn't support dynamic operations

* Enable `PromptGetter` to handle #labels itself

* Add ov inferencer

* Fix overflow during casting dtype & duplicated cast

* Fix

* Add unit&integration tests

* pre-commit

* Fix original vpms

* Fix intg & e2e tests

* Change mo CLI to API

* precommit

* Remove blocks

* Update CHANGELOG.md

* Avoid repeatedly assigning constant tensors/arrays

* Fix typo

* Automate performance benchmark (#2742)

* Add parameterized perf test template

* Split acccuracy / perf tests

* Automate speed test setting

* Add benchmark summary fixture

* Add multi/h-label tests

* Add detection tests

* Add instance segmentationt tests

* Add tiling tests

* Add semantic segmenation tests

* Add anomaly test

* Update tools/expreiment.py (#2751)

* have constant exp directory name

* support to parse dynamic eval output

* align with pre-commit

* fix minor unit test bug

* Add performance benchmark github action workflow (#2762)

* Split accuracy & speed benchmark github workflows (#2763)

* Fix a bug that error is raised when train set size is greater than minimumof batch size in HPO by exactly 1 (#2760)

deal with HPO edge case

* Fix a bug that a process tracking resource usage doesn't exit when main process raises an error (#2765)

* termiate a process tracking resource usage if main process raises an error

* call stop() only if ExitStack is None

* Skip large datasets for iSeg perf benchmark (#2766)

* Support multiple experiments in single recipe for tools/experiment.py (#2757)

* implement draft version

* update logging failed cases

* align with pre-commit

* add doc string

* Update README file

* fix bugs: single command, failed case output

* exclude first epoch from calculating iter time

* fix weird name used when there is no variables

* align with pre-commit

* initialize iter_time and data_time at first

* Enable perf benchmark result logging to mlflow server (#2768)

* Bump datumaro version to 1.6.0rc1 (#2784)

* bump datumaro version to 1.6.0rc1

* remove rust toolchain installation step from workflows

* Update perf logging (#2785)

* Update perf logging workflow to get branch+sha from gh context (#2791)

* update perf logging workflow to get branch+sha from gh context
* skip logging when tracking server uri is not configured

* Add visual prompting zero-shot learning (optimize, documentation, bug fixes) (#2753)

* Fix to resize bbox

* (WIP) Add post-checking between masks with different labels

* Fix to use the first mask in the first loop

* Add post-checking between masks with different labels

* pre-commit

* Add optimize task

* pre-commit

* Add e2e

* Update documentation

* Update CHANGELOG

* Check performance benchmark result with reference (#2821)

* Average 'Small' (/1 /2 /3) dataset benchmark results

* Load perf result with indexing

* Add speed ref check for all tasks

* Add accuracy ref check for all tasks

* Mergeback releases/1.5.0 to develop (#2830)

* Update MAPI version (#2730)

* Update dependency for exportable code (#2732)

* Filter invalid polygon shapes (#2795)

---------

Co-authored-by: Vladislav Sovrasov <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>

* Create OSSF scorecard workflow (#2831)

* Fix ossf/scorecard-action version (#2832)

* Update scorecard.yml

* Update perf benchmark reference (#2843)

* Set default wf permission to read-all (#2882)

* Remedy token permission issue (#2888)

* remedy token-permission issues - part2

* removed dispatch event from scorecard wf

* Add progress callback interface to HPO (#2889)

* add progress callback as HPO argument

* deal with edge case

* Restrict configurable parameters to avoid unreasonable cost for SaaS trial (#2891)

* Reduce max value of POT samples to 1k

* Reduce max value of num_iters to 1k

* Fix pre-commit

* Fix more token-permission issues - part3 (#2893)

* Resolve pinned-dependency issues on publish_internal workflow (#2907)

* Forward unittest workloads to AWS (#2887)

* Resolve pinned dependency issues on workflows (#2909)

* Fix pinned-dependency issues - part2 (#2911)

* Add pinning dependencies (#2916)

* Update pip install cmd to use hashes (#2919)

* Fix HPO progress callback bug (#2908)

fix minor bug

* Fix pinned-dependencies issues (#2929)

* Remove unused test files (#2930)

* Update weekly workflow to run perf tests (#2920)

* update weekly workflow to run perf tests

* Fix missing fixture in perf test

* update input to perf tests for weekly

---------

Co-authored-by: Songki Choi <[email protected]>

* Adjust permission of documentation workflows from pages to contents for writing (#2933)

* remove unused import

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Yunchu Lee <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: Vladislav Sovrasov <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: sungmanc <[email protected]>
eunwoosh added a commit that referenced this pull request Mar 11, 2024
@kprokofi @sovrasov
@goodsong81 @sungchul2 @eunwoosh @yunchu @vinnamkim @negvet @jaegukhyun @sungmanc @eugene123tw @wonjuleee @djdameln @GalyaZalesskaya @harimkang @ashwinvaidya17 @chuneuny-emily
Add support of the Intel dGPUs devices (#3058)
ccbfac1 
* POC: xpu support (#2553)

* Upgrade torch==2.0.1 torchvision==0.15.2 mmcv-full=1.7.1

Signed-off-by: Songki Choi <[email protected]>

* Rollback to mmcv-full==1.7.0 due to mmaction conflict

Signed-off-by: Songki Choi <[email protected]>

* Fix unit test

Signed-off-by: Songki Choi <[email protected]>

* Update cls to run on xpu

* Enable XPU for detection

* Disable FP16 for detection models

* Add xpu data parallel to core utils

* Del copys of XPUDP

* Update sseg XPU support

* Update XPUDP

* added fix for OD and IS. Not debugged

* Upgrade torch==2.0.1 torchvision==0.15.2 mmcv-full=1.7.1

Signed-off-by: Songki Choi <[email protected]>

* Rollback to mmcv-full==1.7.0 due to mmaction conflict

Signed-off-by: Songki Choi <[email protected]>

* Fix unit test

Signed-off-by: Songki Choi <[email protected]>

* Update cls to run on xpu

* Enable XPU for detection

* Disable FP16 for detection models

* Add xpu data parallel to core utils

* Del copys of XPUDP

* Update sseg XPU support

* Update XPUDP

* Fix import of assigner

* Add mmdet ops patching to inference

* Add xpu seed

* Fix linters

* Cleanup in cls train runner

* Cleanup in segm train function

* Fix linters

* Fix linters

* Cleanup

* Update NMS patch

* Use GPU id to move model on target XPU

* Disable fp16 for swinT

* Add XPU device info to OTX env report

* Fix linters

* Update new inits

* Update autocast

* Update fallback to CPU

* Update target device in XPU data parallel

* Revert "Update autocast"

This reverts commit 0ff53ebccd2007fc38e387919e306d476a695635.

* Revert "Patch to enable FP16 on XPU"

This reverts commit fd91f6744f238546c92ae46a7d193369aebc17b4.

* FP16 support for XPU

* Enable FP 16

* Patch to enable FP16 on XPU

* Fix linters

---------

Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: kprokofi <[email protected]>

* Fix tv nms check (#2582)

* POC: HPU support (#2574)

* Add hpu.is_available

* Add hpu in `configure_device`

* Add hpu optimizers

* Update `train_model`

* Add `HPUDataParallel`

* Fix

* Enable HPUOptimizerHooks

* Enable autocast

* Update

* (WIP) Enable dataloader

* (tmp) Disenable habana loader

* Fix to convert bfloat16 tensor to float32 for numpy

* Update `htcore.mark_step` location

* Move setting optimizer hooks into configurer

* Fix to enable bf16

* (WIP) Enable gpu migration

* enable ocr_lite_hrnet_18_mod2 training

* register hpu optimizer once

* detch seg logit for calculating acc only when hpu is used

* change hpu_optimizer path

* cache whether hpu_available

* align with other args

* move hpu_migration into otx.algo.common.util init file

* Refactoring

* move the code changing adam to adamw into mmseg.apis.train.py

* move the code model.to in mmseg.api.train.py

* use use_autocast properly

* remove HPUDistOptimizerHook

* precommit

* Fix unit test

* Use `self.src_device_obj`

* Add function for repeated pattern

* precommit

---------

Co-authored-by: Shin, Eunwoo <[email protected]>

* Merge develop & fix YOLOX inference in BFP16 on XPU (#2602)

Fix yolox with bfp16 dtype on xpu

* Object Detection with Gaudi2 (#2608)

* added support for OD on habana

* added hpu_opt

* added OD support.

* optimize a bit YOLOX. Now, inference is fast. Training still freezes

* SSD, ATSS e2e training

* stabilize mask rcnn a bit

* don't put gt on hpu for OD

* minor fix

* Enable e2e training for Instance Segmentation.

* clean the code stage 1

* clean code 2

* fix pre-commit

* minor

* change cast of bf16

* reply comments

* align with pre-commit

---------

Co-authored-by: eunwoosh <[email protected]>

* Update XPU detection (#2623)

* Add `ModuleCacher` for HPU graphs (#2624)

* Add `ModuleCacher` on classification and segmentation

* Fix

* Update Intel devices branch to the latest develop state (#2666)

* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - https://github.com/tox-dev/tox/issues/3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <[email protected]>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <[email protected]>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <[email protected]>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <[email protected]>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <[email protected]>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <[email protected]>

* Prevent division by zero

Signed-off-by: Songki Choi <[email protected]>

* Update license

Signed-off-by: Songki Choi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <[email protected]>

* Fix inefficient sampling

Signed-off-by: Songki Choi <[email protected]>

* Revert indexing

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

---------

Signed-off-by: Songki Choi <[email protected]>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <[email protected]>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686103df873d020681f6d504f9595ce4a963.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <[email protected]>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Yunchu Lee <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: sungmanc <[email protected]>

* Fix mixed & lower precision training (#2668)

* remove dtype argument in torch.xpu.optimize

* Add `custom_auto_fp16` to use xpu autocast

* Update `forward`s to use `custom_auto_fp16`

* precommit

* Disable FP16 training

* Add `custom_force_fp32`

* Removed what force casting tensors to bf16

* Add `XPUOptimizerHook` and `XPUGradScaler`

* precommit

* Enable lower precision training

* Remove dtype check for lower precision

* Add `bf16_training` in recipe

* fix

* Remove unused module

* Change `XPUOptimizerHook` to `BFp16XPUOptimizerHook`

* Fix for common devices which don't use bf16

* precommit

* Enable to use `auto_fp16` as it is

* Add try-except avoiding mmcv import error

* Fix error type

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Shin, Eunwoo <[email protected]>

* Add XPU support to anomaly task (#2677)

* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - https://github.com/tox-dev/tox/issues/3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit tests

* Fix auto input size mismatch in eval & export (#2530)

* Fix auto input size mismatch in eval & export

* Re-enable E2E tests for Issue#2518

* Add input size check in export testing

* Format float numbers in log

* Fix NNCF export shape mismatch

* Fix saliency map issue

* Disable auto input size if tiling enabled

---------

Signed-off-by: Songki Choi <[email protected]>

* Update ref. fq number for anomaly e2e2 (#2547)

* Skip e2e det tests by issue2548 (#2550)

* Add skip to chained TC for issue #2548 (#2552)

* Update for release 1.4.3 (#2551)

* Update MAPI for 1.5 release (#2555)

Upgrade MAPI to v 0.1.6 (#2529)

* Upgrade MAPI

* Update exp code demo commit

* Fix MAPI imports

* Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Disable QAT for SegNexts (#2565)

* Disable NNCF QAT for SegNext

* Del obsolete pot configs

* Move NNCF skip marks to test commands to avoid duplication

* Add Anomaly modelAPI changes to releases/1.4.0 (#2563)

* bug fix for legacy openvino models

* Apply otx anomaly 1.5 changes

* Fix tests

* Fix compression config

* fix modelAPI imports

* update integration tests

* Edit config types

* Update keys in deployed model

---------

Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>

* Fix the CustomNonLinearClsHead when the batch_size is set to 1 (#2571)

Fix bn1d issue

Co-authored-by: sungmanc <[email protected]>

* Update ModelAPI configuration (#2564 from 1.4) (#2568)

Update ModelAPI configuration (#2564)

* Update MAPI rt infor for detection

* Upadte export info for cls, det and seg

* Update unit tests

* Update for 1.4.4rc1 (#2572)

* Hotfix DatasetEntity.get_combined_subset function loop (#2577)

Fix get_combined_subset function

* Revert default input size to `Default` due to YOLOX perf regression (#2580)

Signed-off-by: Songki Choi <[email protected]>

* Fix for the degradation issue of the classification task (#2585)

* Revert to sync with 1.4.0

* Remove repeat data

* Convert to the RGB value

* Fix color conversion logic

* Fix precommit

* Bump datumaro version to 1.5.1rc3 (#2587)

* Add label ids to anomaly OpenVINO model xml (#2590)

* Add label ids to model xml

---------

* Fix DeiT-Tiny model regression during class incremental training (#2594)

* enable IBloss for DeiT-Tiny

* update changelog

* add docstring

* Add label ids to model xml in release 1.5 (#2591)

Add label ids to model xml

* Fix DeiT-Tiny regression test for release/1.4.0 (#2595)

* Fix DeiT regression test

* update changelog

* temp

* Fix mmcls bug not wrapping model in DataParallel on CPUs (#2601)

Wrap multi-label and h-label classification models by MMDataParallel in case of CPU training.
---------
Signed-off-by: Songki Choi <[email protected]>

* Fix h-label loss normalization issue w/ exclusive label group of singe label (#2604)

* Fix h-label loss normalization issue w/ exclusive label group with signle label

* Fix non-linear version

---------
Signed-off-by: Songki Choi <[email protected]>

* Boost up Image numpy accessing speed through PIL (#2586)

* boost up numpy accessing speed through PIL

* update CHANGELOG

* resolve precommit error

* resolve precommit error

* add fallback logic with PIL open

* use convert instead of draft

* Add missing import pathlib for cls e2e testing (#2610)

* Fix division by zero in class incremental learning for classification (#2606)

* Add empty label to reproduce zero-division error

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

* Fix empty label 4 -> 3

Signed-off-by: Songki Choi <[email protected]>

* Prevent division by zero

Signed-off-by: Songki Choi <[email protected]>

* Update license

Signed-off-by: Songki Choi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Songki Choi <[email protected]>

* Fix inefficient sampling

Signed-off-by: Songki Choi <[email protected]>

* Revert indexing

Signed-off-by: Songki Choi <[email protected]>

* Fix minor typo

Signed-off-by: Songki Choi <[email protected]>

---------

Signed-off-by: Songki Choi <[email protected]>

* Unify logger usage (#2612)

* unify logger

* align with pre-commit

* unify anomaly logger to otx

* change logger file path

* align with pre-commit

* change logger file path in missing file

* configure logger after ConfigManager is initialized

* configure logger when ConfigManager instance is initialized

* update unit test code

* move config_logger to each cli file

* align with pre-commit

* change part still using mmcv logger

* Fix XAI algorithm for Detection (#2609)

* Impove saliency maps algorithm for Detection

* Remove extra changes

* Update unit tests

* Changes for 1 class

* Fix pre-commit

* Update CHANGELOG

* Tighten dependency constraint only adapting latest patches (#2607)

* tighten dependency constratint only adapting latest patches

* adjust scikit-image version w.r.t python version

* adjust tensorboard version w.r.t python version

* remove version specifier for scikit-image

* Add metadata to optimized model (#2618)

* bug fix for legacy openvino models

* Add metadata to optimized model

* Revert formatting changes

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* modify omegaconf version constraint

* [release 1.5.0] Fix XAI algorithm for Detection (#2617)

Update detection XAI algorithm

* Update dependency constraint (#2622)

* Update tpp (#2621)

* Fix h-label bug of missing parent labels in output (#2626)

* Fix h-label bug of missing parent labels in output

* Fix h-label test data label schema

* Update CHANGELOG.md

---------
Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow (#2625)

update publish workflow to push whl to internal pypi

* bump datumaro version to ~=1.5.0

* fixed mistake while mergeing back 1.4.4

* modifiy readme

* remove openvino model wrapper class

* remove openvino model wrapper tests

* [release 1.5.0] DeiT: enable tests + add ViTFeatureVectorHook (#2630)

Add ViT feature vector hook

* Fix docs broken link to datatumaro_h-label

Signed-off-by: Songki Choi <[email protected]>

* Fix wrong label settings for non-anomaly task ModelAPIs

Signed-off-by: Songki Choi <[email protected]>

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

Fix e2e XAI ref value

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* update release note and readme

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

* fix datumaro version to 1.6.0rc0

* Mergeback 1.5.0 to develop (#2642)

* Update publish workflow for tag checking (#2632)

* Update e2e tests for XAI Detection (#2634)

* Disable QAT for newly added models (#2636)

* Update release note and readme (#2637)

* remove package upload step on internal publish wf

* update release note and, changelog, and readme

* update version string to 1.6.0dev

---------

Co-authored-by: Galina Zalesskaya <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>

* Revert "Mergeback 1.5.0 to develop" (#2645)

Revert "Mergeback 1.5.0 to develop (#2642)"

This reverts commit 2f67686103df873d020681f6d504f9595ce4a963.

* Add a tool to help conduct experiments (#2651)

* implement run and experiment

* implement experiment result aggregator

* refactor experiment.py

* refactor run.py

* get export model speed

* add var collumn

* refactor experiment.py

* refine a way to update argument in cmd

* refine resource tracker

* support anomaly on research framework

* refine code aggregating exp result

* bugfix

* make other task available

* eval task save avg_time_per_images as result

* Add new argument to track CPU&GPU utilization and memory usage (#2500)

* add argument to track resource usage

* fix bug

* fix a bug in a multi gpu case

* use total cpu usage

* add unit test

* add mark to unit test

* cover edge case

* add pynvml in requirement

* align with pre-commit

* add license comment

* update changelog

* refine argument help

* align with pre-commit

* add version to requirement and raise an error if not supported values are given

* apply new resource tracker format

* refactor run.py

* support optimize in research framework

* cover edge case

* Handle a case where fail cases exist

* make argparse raise error rather than exit if problem exist

* revert tensorboard aggregator

* bugfix

* save failed cases as yaml file

* deal with integer in variables

* add epoch to metric

* use latest log.json file

* align with otx logging method

* move experiment.py from cli to tools

* refactor experiment.py

* merge otx run feature into experiment.py

* move set_arguments_to_cmd definition into experiment.py

* refactor experiment.py

* bugfix

* minor bugfix

* use otx.cli instead of each otx entry

* add feature to parse single workspace

* add comments

* fix bugs

* align with pre-commit

* revert parser argument

* align with pre-commit

* Revert inference batch size to 1 for instance segmentation (#2648)

Signed-off-by: Songki Choi <[email protected]>

* Remove unnecessary log while building a model (#2658)

* revert logger in otx/algorithms/detection/adapters/mmdet/utils/builder.py

* revert logger in otx/algorithms/classification/adapters/mmcls/utils/builder.py

* make change more readable

* Fix a minor bug of experiment.py (#2662)

fix bug

* Not check avg_time_per_image during test (#2665)

* ignore avg_time_per_image during test

* do not call stdev when length of array is less than 2

* ignore avg_time_per_image during regerssion test

* Update device selection logic in classificaiton

* Add xpu accelerator

* Tmp patch for anomaly trainer

* Use XPU callback for anomaly training

* Update xpu accelerator

* Fix for anomaly xpu callback

* Fix validation batch logic

* Cleanup, add docstrings

* Refine xpu callback

---------

Signed-off-by: Kim, Vinnam <[email protected]>
Signed-off-by: Songki Choi <[email protected]>
Co-authored-by: Yunchu Lee <[email protected]>
Co-authored-by: Kim, Sungchul <[email protected]>
Co-authored-by: Vinnam Kim <[email protected]>
Co-authored-by: Evgeny Tsykunov <[email protected]>
Co-authored-by: Songki Choi <[email protected]>
Co-authored-by: Eunwoo Shin <[email protected]>
Co-authored-by: Jaeguk Hyun <[email protected]>
Co-authored-by: Sungman Cho <[email protected]>
Co-authored-by: Eugene Liu <[email protected]>
Co-authored-by: Wonju Lee <[email protected]>
Co-authored-by: Dick Ameln <[email protected]>
Co-authored-by: sungchul.kim <sungchul@ikvensx010>
Co-authored-by: GalyaZalesskaya <[email protected]>
Co-authored-by: Harim Kang <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: Ashwin Vaidya <[email protected]>
Co-authored-by: sungmanc <[email protected]>

* Disable mixed precision training on XPU (#2683)

* disable mixed precision training on XPU

* fix optimize error

* refactor XPUDataParallel

* align with pre-commit

* align with pre-commit

* fix bug

* Update anomaly XPU integration (#2697)

* Update anomaly XPU integration

* Update strategy and accelerator

* Cleanup in strategy

* Fix mypy

* remove XPU callback

* Add XPU mixed precision plugin for lightning (#2714)

* Update anomaly XPU integration

* Update strategy and accelerator

* Cleanup in strategy

* Fix mypy

* remove XPU callback

* Add XPU mixed precision lightning training

* Fix linters

* Handle default plugins value

* Update code to support other features than 'train' on XPU (#2704)

* move where patching code for xpu in detection

* implement xpu resource manager in HPO

* consider xpu during adaptive_num_workers

* consider xpu while deciding asynchronous_sha

* align with pre-commit

* change variable name to proper one

* change patching code into configure_device

* update unit test code

* align with pre-commit

* Merge develop to develop-idev (#2727)

* Update base.txt

updated dependency version of datumaro

* Update __init__.py

update version string

* Update requirements.txt

* Temporarily skip visual prompting openvino integration test (#2323)

* Fix import dm.DatasetSubset (#2324)

Signed-off-by: Kim, Vinnam <[email protected]>

* Fix semantic segmentation soft prediction dtype (#2322)

* Fix semantic segmentation soft prediction dtype

* relax ref sal vals check

---------

Co-authored-by: Songki Choi <[email protected]>

* Contrain yapf verison lesser than 0.40.0 (#2328)

contrain_yapf_version

* Fix detection e2e tests (#2327)

Fix for detection

* Mergeback: Label addtion/deletion 1.2.4 --> 1.4.0 (#2326)

* Make black happy

* Fix conflicts

* Merge-back: add test datasets and edit the test code

* Make black happy

* Fix mis-merge

* Make balck happy

* Fix typo

* Fix typoi

---------

Co-authored-by: Songki Choi <[email protected]>

* Bump datumaro up to 1.4.0rc2 (#2332)

bump datumaro up to 1.4.0rc2

* Tiling Doc for releases 1.4.0 (#2333)

* Add tiling documentation

* Bump otx version to 1.4.0rc2 (#2341)

* OTX deploy for visual prompting task  (#2311)

* Enable `otx deploy`

* (WIP) integration test

* Docstring

* Update args for create_model

* Manually set image embedding layout

* Enable to use model api for preprocessing
- `fit_to_window` doesn't work expectedly, so newly implemented `VisualPromptingOpenvinoAdapter` to use new resize function

* Remove skipped test

* Updated

* Update unit tests on model wrappers

* Update

* Update configuration

* Fix not to patch pretrained path

* pylint & update model api version in docstring

---------

Co-authored-by: Wonju Lee <[email protected]>

* Bump albumentations version in anomaly requirements (#2350)

increment albumentations version

* Update action detection (#2346)

* Remove skip mark for PTQ test of action detection

* Update action detection documentation

* Fix e2e (#2348)

* Change classification dataset from dummy to toy

* Revert test changes

* Change label name for multilabel dataset

* Revert e2e test changes

* Change ov test cases' threshold

* Add parent's label

* Update ModelAPI in 1.4 release (#2347)

* Upgrade model API

* Update otx in exportable code

* Fix unit tests

* Fix black

* Fix detection inference

* Fix det tiling

* Fix mypy

* Fix demo

* Fix visualizer in demo

* Fix black

* Add OTX optimize for visual prompting task (#2318)

* Initial commit

* Update block

* (WIP) otx optimize

* Fix

* WIP

* Update configs & exported outputs

* Remove unused modules for torch

* Add unit tests

* pre-commit

* Update CHANGELOG

* Update detection docs (#2335)

* Update detection docs

* Revert template id changes

* Fix wrong template id

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

* Update docs/source/guide/explanation/algorithms/object_detection/object_detection.rst

Co-authored-by: Eunwoo Shin <[email protected]>

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Add visual prompting documentation (#2354)

* (WIP) write docs

* Add visual prompting documentation

* Update CHANGELOG

---------

Co-authored-by: sungchul.kim <sungchul@ikvensx010>

* Remove custom modelapi patch in visual prompting (#2359)

* Remove custom modelapi patch

* Update test

* Fix graph metric order and label issues (#2356)

* Fix graph metric going backward issue
* Add license notice
* Fix pre-commit issue
* Add rename items & logic for metric
---------
Signed-off-by: Songki Choi <[email protected]>

* Update multi-label document and conversion script (#2358)

Update docs, label convert script

* Update third party programs (#2365)

* Make anomaly task compatible with older albumentations versions (#2363)

* fix transforms export in metadata

* wrap transform dict

* add todo for updating to_dict call

* Fixing detection saliency map for one class case (#2368)

* fix softmax

* fix validity tests

* Add e2e test for visual prompting (#2360)

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* Delete unused configuration.yaml

* Edit test_name

* Add to limit activation range

* Update from `vp` to `visprompt`

* Fix about no returning the first label

* pre-commit

* (WIP) otx optimize

* pre-commit

* (WIP) set e2e

* Remove nncf config

* Add visual prompting requirement

* Add visual prompting in tox

* Add visual prompting in setup.py

* Fix typo

* pre-commit

* Add actions

* Update tests/e2e/cli/visual_prompting/test_visual_prompting.py

Co-authored-by: Jaeguk Hyun <[email protected]>

* Skip PTQ e2e test

* Change task name

* Remove skipped tc

---------

Co-authored-by: Jaeguk Hyun <[email protected]>

* Fix e2e (#2366)

* Change e2e reference name

* Update openvino eval threshold for multiclass classification

* Change comment message

* Fix tiling e2e tests

---------

Co-authored-by: GalyaZalesskaya <[email protected]>

* Add Dino head unit tests (#2344)

Recover DINO head unit tests

* Update for release 1.4.0rc2 (#2370)

* update for release 1.4.0rc2

* Add skip mark for unstable unit tests

---------

Co-authored-by: jaegukhyun <[email protected]>

* Fix NNCF training on CPU (#2373)

* Align label order between Geti and OTX (#2369)

* align label order

* align with pre-commit

* update CHANGELOG.md

* deal with edge case

* update type hint

* Remove CenterCrop from Classification test pipeline and editing missing docs link (#2375)

* Fix missing link for docs and removing centercrop for classification data pipeline

* Revert the test threshold

* Fix H-label classification (#2377)

* Fix h-labelissue

* Update unit tests

* Make black happy

* Fix unittests

* Make black happy

* Fix update heades information func

* Update the logic: consider the loss per batch

* Update for release 1.4 (#2380)

* updated for 1.4.0rc3

* update changelog & release note

* bump datumaro version up

---------

Co-authored-by: Songki Choi <[email protected]>

* Switch to PTQ for sseg (#2374)

* Switch to PTQ for sseg

* Update log messages

* Fix invalid import structures in otx.api (#2383)

Update tiler.py

* Update for 1.4.0rc4 (#2385)

update for release 1.4.0rc4

* [release 1.4.0] XAI: Return saliency maps for Mask RCNN IR async infer (#2395)

* Return saliency maps for openvino async infer

* add workaround to fix yapf importing error

---------

Co-authored-by: eunwoosh <[email protected]>

* Update for release 1.4.0 (#2399)

update version string

Co-authored-by: Sungman Cho <[email protected]>

* Fix broken links in documentation (#2405)

* fix docs links to datumaro's docs
* fix docs links to otx's docs
* bump version to 1.4.1

* Update exportable code README (#2411)

* Updated for release 1.4.1 (#2412)

updated for release 1.4.1

* Add workaround for the incorrect meta info M-RCNN (used for XAI) (#2437)

Add workaround for the incorrect mata info

* Add model category attributes to model template (#2439)

Add model category attributes to model template

* Add model category & status fields in model template

* Add is_default_for_task attr to model template

* Update model templates with category attrs

* Add integration tests for model templates consistency

* Fix license & doc string

* Fix typo

* Refactor test cases

* Refactor common tests by generator

---------
Signed-off-by: Songki Choi <[email protected]>

* Update for 1.4.2rc1 (#2441)

update for release 1.4.2rc1

* Fix label list order for h-label classification (#2440)

* Fix label list for h-label cls
* Fix unit tests

* Modified fq numbers for lite HRNET (#2445)

modified fq numbers for lite HRNET

* Update PTQ ignored scope for hrnet 18  mod2 (#2449)

Update ptq ignored scope for hrnet 18  mod2

* Fix OpenVINO inference for legacy models (#2450)

* bug fix for legacy openvino models

* Add tests

* Specific exceptions

---------

* Update for 1.4.2rc2 (#2455)

update for release 1.4.2rc2

* Prevent zero-sized saliency map in tiling if tile size is too big (#2452)

* Prevent zero-sized saliency map in tiling if tile size is too big

* Prevent zero-sized saliency in tiling (PyTorch)

* Add unit tests for Tiler merge features methods

---------

Co-authored-by: Galina <[email protected]>

* Update pot fq reference number (#2456)

update pot fq reference number to 15

* Bump datumaro version to 1.5.0rc0 (#2470)

bump datumaro version to 1.5.0rc0

* Set tox version constraint (#2472)

set tox version constraint - https://github.com/tox-dev/tox/issues/3110

* Bug fix for albumentations (#2467)

* bug fix for legacy openvino models

* Address albumentation issue

---------

Co-authored-by: Ashwin Vaidya <[email protected]>

* update for release 1.4.2rc3

* Add a dummy hierarchical config required by MAPI (#2483)

* bump version to 1.4.2rc4

* Bump datumaro version (#2502)

* bump datumaro version

* remove deprecated/reomved attribute usage of the datumaro

* Upgrade nncf version for 1.4 release (#2459)

* Upgrade nncf version

* Fix nncf interface warning

* Set the exact nncf version

* Update FQ refs after NNCF upgrade

* Use NNCF from pypi

* Update version for release 1.4.2rc5 (#2507)

update version for release 1.4.2rc5

* Update for 1.4.2 (#2514)

update for release 1.4.2

* create branch release/1.5.0

* Delete mem cache handler after training is done (#2535)

release mem cache handler after training is done

* Fix bug that auto batch size doesn't consider distributed training (#2533)

* consider distributed training while searching batch size

* update unit test

* reveret gpu memory upper bound

* fix typo

* change allocated to reserved

* add unit test for distributed training

* align with pre-commit

* Apply fix progress hook to release 1.5.0 (#2539)

* Fix hook's ordering issue. AdaptiveRepeatHook changes the runner.max_iters before the ProgressHook

* Change the expression

* Fix typo

* Fix multi-label, h-label issue

* Fix auto_bs issue

* Apply suggestions from code review

Co-authored-by: Eunwoo Shin <[email protected]>

* Reflecting reviews

* Refactor the name of get_data_cfg

* Revert adaptive hook sampler init

* Refactor the function name: get_data_cfg -> get_subset_data_cfg

* Fix unit test errors

* Remove adding AdaptiveRepeatDataHook for autobs

* Remove unused import

* Fix detection and segmentation case in Geti scenario

---------

Co-authored-by: Eunwoo Shin <[email protected]>

* Re introduce adaptive scheduling for training (#2541)

* Re-introduce adaptive patience for training

* Revert unit tests

* Update for release 1.4.3rc1 (#2542)

* Mirror Anomaly ModelAPI changes (#2531)

* Migrate anomaly exportable code to modelAPI (#2432)

* Fix license in PR template

* Migrate to modelAPI

* Remove color conversion in streamer

* Remove reverse_input_channels

* Add float

* Remove test as metadata is no longer used

* Remove metadata from load method

* remove anomalib openvino inferencer

* fix signature

* Support logacy OpenVINO model

* Transform image

* add configs

* Re-introduce adaptive training (#2543)

* Re-introduce adaptive patience for training

* Revert unit t…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sungmanc sungmanc sungmanc approved these changes

@yunchu yunchu yunchu approved these changes

@wonjuleee wonjuleee Awaiting requested review from wonjuleee

@eunwoosh eunwoosh Awaiting requested review from eunwoosh

@jaegukhyun jaegukhyun Awaiting requested review from jaegukhyun

Assignees
No one assigned
Labels
ALGO Any changes in OTX Algo Tasks implementation BUILD DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM TEST Any changes in tests
Projects
None yet
Milestone
1.4.0
Development

Successfully merging this pull request may close these issues.
5 participants
@sungchul2 @jaegukhyun @yunchu @sungmanc @eunwoosh